Grounding spoken interaction with real-time gaze in dynamic virtual environments

Gaze is an important cue in visually situated dialog, grounding referring expressions to objects in the environment. We present a new technique which demonstrates that monitoring real-time listener gaze – and giving appropriate feedback – enhances reference resolution by the listener: In a 3D virtual environment, users followed directional instructions, including pressing a number of buttons that were identified using referring expression generated by the system (see GIVE; Koller et al., 2010). Gaze to the intended referent following a referring expression was taken as evidence of successful understanding and elicited positive feedback; by contrast, gaze to other objects triggered early negative feedback. We compared this eye movement-based feedback strategy with two baseline systems, revealing that the eye-movement based feedback leads to significantly more successful button presses than the other two strategies. Our findings suggest that listener gaze immediately following a referring expression reliably indicates how a listener resolved the expression.